Explainable AI, Homework 3

LIME analysis for the heart disease dataset.

Example LIME decompositions

Below, we can see LIME decompostions for an XGBoost model. The two samples were previously analysed in Homework 2.

alt text

alt text

LIME stability

XGBoost sample

It seems that the LIME values are quite stable for the XGBoost model.

Below, we can see two decompositions for the same sample, picked as the most different ones from a sample of 20.

For all the sampled explanations, the set of top 3 variables remained unchanged. The values don't change much, there is no cases of the same variable getting different signs. The IOU between the sets of top 10 predictors is 0.82, with just one variable being different -- thalachh and restecg_1, which are both in the lower part of the top 10.

We can conclude that, if the values are of an order of magnitude that makes them meaningful, sampling does not change much in the results.

Logistic regression extreme sample

For the logistic regression model, the differences between runs are much more visible.

Below, we can see two explanations for the same and the same sample. The second explanation was chosen as the one with the biggest absolute value of an explanation among the entire test set. As such, it is in a sense an outlier, and the differences between explanations represent an extreme case, not an average one. We can also note that the values are much smaller than in the case of the XGBoost model, which makes them less meaningful and reliable.

With that noted, we can see that the differences here are extreme:

  • The values are 8 orders of magnitude different.
  • The IOU between the top 10 predictor sets is just 0.33.
  • Variables with differing contribution signs are present: sex_1 and restecg_1.

LIME and SHAP comparison

Below, we can see the LIME and SHAP analyses for a selected sample.

We can see that the rankings of variables differ. The sets of top variables are similar, with an IOU of 0.58, meaning that just 2 of the top values for SHAP are not included in the LIME explanation. There are no contradictions between the signs of the two explanations -- for variables that appear in both graphs, the signs of their contributions agree.

Below, we have the explanations for the other previously chosen sample. Again, altghough the ordering of the features differs, there are no variables that have different signs in the two explanations. The IOU is 0.73, with just one variable from SHAP not appearing in LIME (exng_1).

Taking into account the variability of LIME explanations, we can conclude that, although the values can vary considerably in unlucky cases, in an average case we can expect the two methods to yield reasonably similar results, with no contradictions. The main difference here is the scale of the values, and the interpretability of the two methods, with the LIME results not summing up to a meaningful value.

LIME values between two models

Even from the example above in the stability section, which has been sampled for the same observation for the two models, we can see that the LIME values vary considerably between the two models.

I have gathered LIME explanations for the enitre test set (61 samples) for two models: XGBoost and a logistic regression. Below, we can see a summary of two dataframes, collecting information about the maximal absolute result values for each sample -- first for the XGBoost model, and then for the logistic regression.

We can see that the order of magnitude of the values for the two models is different. LIME seems to not be able to find much of an explanation for the logistic regression model, with the values being near zero.

image.png

image.png

Appendix

In [1]:
import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
import xgboost
import dalex as dx
import plotly.express as px
from scipy.stats import zscore

import warnings
warnings.filterwarnings("ignore")
In [2]:
DATA_PATH = "../heart.csv"
CATEGORICAL_COLUMNS = ['sex', 'cp', 'fbs', 'restecg', 'caa', 'exng', 'slp', 'thall']
TARGET_COLUMN = 'output'

Read in and prepare data

In [3]:
df = pd.read_csv(DATA_PATH)
df.head()
Out[3]:
age sex cp trtbps chol fbs restecg thalachh exng oldpeak slp caa thall output
0 63 1 3 145 233 1 0 150 0 2.3 0 0 1 1
1 37 1 2 130 250 0 1 187 0 3.5 0 0 2 1
2 41 0 1 130 204 0 0 172 0 1.4 2 0 2 1
3 56 1 1 120 236 0 1 178 0 0.8 2 0 2 1
4 57 0 0 120 354 0 1 163 1 0.6 2 0 2 1
In [4]:
df.describe()
Out[4]:
age sex cp trtbps chol fbs restecg thalachh exng oldpeak slp caa thall output
count 303.000000 303.000000 303.000000 303.000000 303.000000 303.000000 303.000000 303.000000 303.000000 303.000000 303.000000 303.000000 303.000000 303.000000
mean 54.366337 0.683168 0.966997 131.623762 246.264026 0.148515 0.528053 149.646865 0.326733 1.039604 1.399340 0.729373 2.313531 0.544554
std 9.082101 0.466011 1.032052 17.538143 51.830751 0.356198 0.525860 22.905161 0.469794 1.161075 0.616226 1.022606 0.612277 0.498835
min 29.000000 0.000000 0.000000 94.000000 126.000000 0.000000 0.000000 71.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
25% 47.500000 0.000000 0.000000 120.000000 211.000000 0.000000 0.000000 133.500000 0.000000 0.000000 1.000000 0.000000 2.000000 0.000000
50% 55.000000 1.000000 1.000000 130.000000 240.000000 0.000000 1.000000 153.000000 0.000000 0.800000 1.000000 0.000000 2.000000 1.000000
75% 61.000000 1.000000 2.000000 140.000000 274.500000 0.000000 1.000000 166.000000 1.000000 1.600000 2.000000 1.000000 3.000000 1.000000
max 77.000000 1.000000 3.000000 200.000000 564.000000 1.000000 2.000000 202.000000 1.000000 6.200000 2.000000 4.000000 3.000000 1.000000
In [5]:
for column in CATEGORICAL_COLUMNS:
    df[column] = df[column].astype(str)

Encode categorical features

In [6]:
df = pd.get_dummies(df, drop_first=True)
df.columns
Out[6]:
Index(['age', 'trtbps', 'chol', 'thalachh', 'oldpeak', 'output', 'sex_1',
       'cp_1', 'cp_2', 'cp_3', 'fbs_1', 'restecg_1', 'restecg_2', 'exng_1',
       'slp_1', 'slp_2', 'caa_1', 'caa_2', 'caa_3', 'caa_4', 'thall_1',
       'thall_2', 'thall_3'],
      dtype='object')

Select and normalise X and y

In [7]:
X = df.drop(columns = [TARGET_COLUMN])
In [8]:
y = df[TARGET_COLUMN]
In [9]:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

Train logistic regression

In [10]:
scaler = StandardScaler().fit(X_train)
X_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
In [11]:
logistic_regression = LogisticRegression().fit(X_scaled, y_train)

Train XGBoost

In [12]:
xgboost_model = xgboost.XGBClassifier(
    n_estimators=200, 
    max_depth=4, 
    use_label_encoder=False, 
    eval_metric="logloss"
)
In [13]:
xgboost_model.fit(X_train, y_train)
Out[13]:
XGBClassifier(base_score=0.5, booster='gbtree', callbacks=None,
              colsample_bylevel=1, colsample_bynode=1, colsample_bytree=1,
              early_stopping_rounds=None, enable_categorical=False,
              eval_metric='logloss', gamma=0, gpu_id=-1,
              grow_policy='depthwise', importance_type=None,
              interaction_constraints='', learning_rate=0.300000012,
              max_bin=256, max_cat_to_onehot=4, max_delta_step=0, max_depth=4,
              max_leaves=0, min_child_weight=1, missing=nan,
              monotone_constraints='()', n_estimators=200, n_jobs=0,
              num_parallel_tree=1, predictor='auto', random_state=0,
              reg_alpha=0, reg_lambda=1, ...)
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

1. Calculate the predictions for some selected observations

The selected model here is XGBoost.

In [14]:
k = 20
x1, y1 = X_test.iloc[0:1], y_test.iloc[0:1]
x2, y2 = X_test.iloc[k:k+1], y_test.iloc[k:k+1]
In [15]:
for x, y in zip((x1, x2), (y1, y2)):
    print(y)
    print(xgboost_model.predict_proba(x))
    print(xgboost_model.predict(x))
179    0
Name: output, dtype: int64
[[0.9927471  0.00725288]]
[0]
267    0
Name: output, dtype: int64
[[0.00563395 0.99436605]]
[1]

2. Calculate the decomposition of these predictions with LIME using the package of choice

The selected package is dalex.

In [16]:
explainer = dx.Explainer(xgboost_model, X_train, y_train)
Preparation of a new explainer is initiated

  -> data              : 242 rows 22 cols
  -> target variable   : Parameter 'y' was a pandas.Series. Converted to a numpy.ndarray.
  -> target variable   : 242 values
  -> model_class       : xgboost.sklearn.XGBClassifier (default)
  -> label             : Not specified, model's class short name will be used. (default)
  -> predict function  : <function yhat_proba_default at 0x7f1e3f86b9d0> will be used (default)
  -> predict function  : Accepts pandas.DataFrame and numpy.ndarray.
  -> predicted values  : min = 1.82e-05, mean = 0.545, max = 1.0
  -> model type        : classification will be used (default)
  -> residual function : difference between y and yhat (default)
  -> residuals         : min = -0.131, mean = -1.02e-05, max = 0.107
  -> model_info        : package xgboost

A new explainer has been created!
In [17]:
for sample in (x1, x2):
    explanation = explainer.predict_surrogate(sample)
    explanation.plot()

3. Compare LIME for various observations in the dataset. How stable are these explanations?

In [18]:
num_samples = 5
X_sample = [X_test.iloc[i:i+1] for i in range(num_samples)]
y_sample = [y_test.iloc[i:i+1] for i in range(num_samples)]
In [19]:
for sample in X_sample:
    explanation = explainer.predict_surrogate(sample)
    explanation.plot()

4. Compare LIME with the explanations obtained using SHAP. What are the main differences between them?

In [20]:
pf_xgboost_classifier_default = lambda m, d: m.predict_proba(d)[:, 1]

shap_explainer = dx.Explainer(xgboost_model, X_train, predict_function=pf_xgboost_classifier_default, label="XGBoost")
Preparation of a new explainer is initiated

  -> data              : 242 rows 22 cols
  -> target variable   : Not specified!
  -> model_class       : xgboost.sklearn.XGBClassifier (default)
  -> label             : XGBoost
  -> predict function  : <function <lambda> at 0x7f1e154cc4c0> will be used
  -> predict function  : Accepts pandas.DataFrame and numpy.ndarray.
  -> predicted values  : min = 1.82e-05, mean = 0.545, max = 1.0
  -> model type        : classification will be used (default)
  -> residual function : difference between y and yhat (default)
  -> model_info        : package xgboost

A new explainer has been created!
In [21]:
sample = x1
explainer.predict_surrogate(sample).plot()
shap_explainer.predict_parts(sample, type="shap").plot()
In [22]:
sample = x2
explainer.predict_surrogate(sample).plot()
shap_explainer.predict_parts(sample, type="shap").plot()

5. Compare LIME between at least two different models. Are there any systematic differences across many observations?

In [23]:
logistic_explainer = dx.Explainer(logistic_regression, X_train, y_train)
Preparation of a new explainer is initiated

  -> data              : 242 rows 22 cols
  -> target variable   : Parameter 'y' was a pandas.Series. Converted to a numpy.ndarray.
  -> target variable   : 242 values
  -> model_class       : sklearn.linear_model._logistic.LogisticRegression (default)
  -> label             : Not specified, model's class short name will be used. (default)
  -> predict function  : <function yhat_proba_default at 0x7f1e3f86b9d0> will be used (default)
  -> predict function  : Accepts pandas.DataFrame and numpy.ndarray.
  -> predicted values  : min = 3.01e-87, mean = 4.32e-20, max = 1e-17
  -> model type        : classification will be used (default)
  -> residual function : difference between y and yhat (default)
  -> residuals         : min = -8.53e-27, mean = 0.545, max = 1.0
  -> model_info        : package sklearn

A new explainer has been created!
In [24]:
for sample in (x1, x2):
    explanation = logistic_explainer.predict_surrogate(sample)
    explanation.plot()
In [25]:
num_samples = len(X_test)
X_sample = [X_test.iloc[i:i+1] for i in range(num_samples)]
y_sample = [y_test.iloc[i:i+1] for i in range(num_samples)]
In [26]:
xgboost_results = []
logistic_results = []
xgboost_expls = {}
logistic_expls = {}

for sample in X_sample:
    idx = sample.index[0]
    expl = explainer.predict_surrogate(sample)
    res = expl.result
    res["sample_id"] = idx
    xgboost_results.append(res)
    xgboost_expls[idx] = expl
    expl = logistic_explainer.predict_surrogate(sample)
    res = expl.result
    res["sample_id"] = idx
    logistic_results.append(res)
    logistic_expls[idx] = expl
In [27]:
xgboost_results = pd.concat(xgboost_results)
logistic_results = pd.concat(logistic_results)
In [28]:
xgboost_results["abs_effect"] = np.abs(xgboost_results.effect)
logistic_results["abs_effect"] = np.abs(logistic_results.effect)
In [29]:
fig = px.histogram(x=xgboost_results.effect)
fig.show()
In [30]:
fig = px.histogram(x=xgboost_results.groupby("sample_id").abs_effect.max())
fig.show()
In [31]:
fig = px.histogram(x=logistic_results.groupby("sample_id").abs_effect.max())
fig.show()
In [32]:
fig = px.histogram(data_frame=logistic_results, x="effect")
fig.show()
In [33]:
xgboost_results.describe()
Out[33]:
effect sample_id abs_effect
count 610.000000 610.000000 610.000000
mean 0.016321 152.311475 0.165027
std 0.175153 90.644570 0.060554
min -0.303506 0.000000 0.077345
25% -0.140655 70.000000 0.114483
50% 0.097442 151.000000 0.147817
75% 0.155994 236.000000 0.212060
max 0.295306 302.000000 0.303506
In [34]:
logistic_results.describe()
Out[34]:
effect sample_id abs_effect
count 6.100000e+02 610.000000 6.100000e+02
mean 8.801998e-19 152.311475 2.987776e-17
std 1.769040e-16 90.644570 1.743607e-16
min -1.968577e-15 0.000000 5.167981e-22
25% -5.346346e-20 70.000000 6.091949e-21
50% 9.918815e-22 151.000000 8.501602e-20
75% 1.339017e-19 236.000000 6.333045e-19
max 1.651157e-15 302.000000 1.968577e-15
In [45]:
xgboost_results.groupby("sample_id").abs_effect.max().describe()
Out[45]:
count    61.000000
mean      0.282335
std       0.008381
min       0.266783
25%       0.275489
50%       0.282575
75%       0.289424
max       0.303506
Name: abs_effect, dtype: float64
In [36]:
logistic_results.groupby("sample_id").abs_effect.max().describe()
Out[36]:
count    6.100000e+01
mean     6.585545e-17
std      3.262401e-16
min      1.139013e-21
25%      1.258997e-20
50%      1.773873e-19
75%      2.850359e-18
max      1.968577e-15
Name: abs_effect, dtype: float64
In [37]:
df = logistic_results.groupby("sample_id").max()
df[df.abs_effect == df.abs_effect.max()]
Out[37]:
variable effect abs_effect
sample_id
286 thall_3 <= 0.00 4.990815e-16 1.968577e-15
In [38]:
logistic_expls[160].plot()
In [39]:
x_max = X_test[X_test.index == 160]
In [40]:
res = logistic_explainer.predict_surrogate(x_max)
res.plot()
In [41]:
res = explainer.predict_surrogate(x_max)
res.plot()
In [42]:
res = explainer.predict_surrogate(x_max)
res.plot()
In [ ]:
 
In [43]:
x_max
Out[43]:
age trtbps chol thalachh oldpeak sex_1 cp_1 cp_2 cp_3 fbs_1 ... exng_1 slp_1 slp_2 caa_1 caa_2 caa_3 caa_4 thall_1 thall_2 thall_3
160 56 120 240 169 0.0 1 1 0 0 0 ... 0 0 0 0 0 0 0 0 1 0

1 rows × 22 columns

In [44]:
res.result
Out[44]:
variable effect
0 0.00 < thall_2 <= 1.00 0.297898
1 caa_1 <= 0.00 0.238026
2 cp_2 <= 0.00 -0.185419
3 thall_3 <= 0.00 0.139740
4 211.25 < chol <= 240.00 0.135459
5 0.00 < sex_1 <= 1.00 -0.119886
6 caa_2 <= 0.00 0.119436
7 exng_1 <= 0.00 0.111471
8 cp_3 <= 0.00 -0.110786
9 0.00 < restecg_1 <= 1.00 0.091776